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We consider a dynamic control problem associated with a gener- 
alized Brownian network, the objective being to minimize expected 
discounted cost over an infinite planning horizon. In this Brownian 
control problem (BCP), both the system manager's control and the 
associated cumulative cost process may be locally of unbounded vari- 
ation. Due to this aspect of the cost process, both the precise state- 
ment of the problem and its analysis involve delicate technical issues. 
We show that the BCP is equivalent, in a certain sense, to a reduced 
Brownian control problem (RBCP) of lower dimension. The RBCP 
is a singular stochastic control problem, in which both the controls 
and the cumulative cost process are locally of bounded variation. 

1. Introduction. The object of study in this paper is a stochastic system 
model that was described in Section 2 of [11] and there called a "generalized 
Brownian network." In this paper we formulate a control problem for that 
model and prove that it is equivalent to a simpler control problem of lower 
dimension. 

In a prior work, Harrison and Van Mieghem [13] provided a similar de- 
velopment for a class of Brownian networks described in [9] and [8]. The 
Brownian network model considered here differs from that considered in 
[13] in two significant respects. 

First, here the state space is a suitable compact convex subset of a Eu- 
clidean space (e.g., a bounded convex polyhedron), whereas it is the positive 

Received June 2004; revised January 2005. 

Supported in part by NSF Grants DMS-00-71408, DMS-03-05272, and a John Simon 
Guggenheim Fellowship. During part of the period of this research, R. J. Williams was sup- 
ported by the Operations, Information and Technology Program of the Graduate School of 
Business, Stanford University, and by the Institute for Mathematics and Its Applications, 
Minneapolis, Minnesota. 

AMS 2000 subject classifications. 60J60, 60K30, 90B15, 90B36. 

Key words and phrases. Stochastic control, singular control, Brownian network model, 
reflected Brownian motion, workload, no-arbitrage, state space collapse, continuous selec- 
tion. 

This is an electronic reprint of the original article published by the 
Institute of Mathematical Statistics in The Annals of Applied Probability, 
2005, Vol. 15, No. 4, 2255-2295. This reprint differs from the original in 
pagination and typographic detail. 



1 



2 



J. M. HARRISON AND R. J. WILLIAMS 



orthant of such a space in [13]. Our restriction to a bounded state space will 
be discussed further below. 

Second, in this paper the process representing the cumulative cost of 
control need not be locally of bounded variation, whereas in [13] it is as- 
sumed to be a nondecreasing process. The utility of this more general cost 
structure was explained and illustrated in [11]. Readers will see that this 
generalized cost structure leads to substantial new difficulties, not just mi- 
nor technical complications. Furthermore, our formulation differs in certain 
technical respects from that in [13], and those differences enable a more sat- 
isfactory mathematical development. In particular, the "weak formulation" 
of admissible controls that we employ in this paper is designed for ease of 
use by researchers who develop heavy traffic limit theorems to justify Brow- 
nian network approximations. Also, by making separate statements about 
a reduced Brownian network and a reduced Brownian control problem, we 
provide a clearer and more complete picture than in [13]. Finally, relative 
to the treatment in [13], here the algebraic manipulations are more exten- 
sive and have a geometric flavor, and we correct an error in [13] related to 
continuous selections (see Appendix A. 3 below). 

The restriction to a bounded state space is essential in our context, as 
noted in Section 9 of [11]. If an unbounded state space were allowed, then 
additional care would be needed to ensure a meaningful formulation, the 
potential problem being that the cumulative cost of control might be un- 
bounded below over a finite time interval. The associated issues have not 
been explored to date. By restricting attention to bounded state spaces, we 
rule out heavy traffic limits of "open" queueing networks in which storage 
buffers have unlimited capacity. However, our model can be used to ap- 
proximate an open queueing network with large finite buffers, and such a 
formulation is arguably more realistic in many application contexts. 

Generalized Brownian networks arise as diffusion approximations for con- 
ventional stochastic processing networks in various application contexts. 
That motivation for the model class has been developed earlier in [11] and 
[10], so it need not be repeated here. Similarly, readers may consult [13] for 
a detailed account of earlier work on the reduction of Brownian networks, 
or to be more precise, reduction of their associated control problems, to 
"equivalent workload formulations" of lower dimension. This kind of model 
reduction is important for purposes of both structural insight and reduced 
computational complexity. Much of what is said in [13] applies equally well 
to the larger model class considered here, and the examples offered in that 
paper illustrate well the character and value of workload reductions. Here 
we simply proceed with the generalized mathematical development, noting 
new effects as they arise. 

The paper is organized as follows. Section 2 lays out the definition of a 
generalized Brownian network, which differs slightly from the model formu- 
lation proposed in [11] in that a "weak formulation" is used here. In Section 
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3 an associated Brownian control problem is formulated. The cost functional 
used here is expected discounted cost over an infinite planning horizon. Some 
care is required in justifying the use of this cost functional, because it is not 
a priori clear that a limit of the finite horizon discounted cost exists as the 
time horizon recedes to infinity. However, the technical Lemma 3.2, proved 
in Appendix A. 2, shows that the cost functional is well defined with values 
in (—00,00]. In Section 4 some algebraic manipulations are performed as 
a precursor to our definitions of a reduced Brownian network and reduced 
Brownian control problem. Section 5 establishes the equivalence, in a certain 
sense, of the generalized Brownian network and a reduced Brownian network 
of lower dimension. We complete our mathematical development in Section 
6 by showing that the Brownian control problem formulated in Section 3 for 
the generalized Brownian network is equivalent to a simpler control problem, 
called the reduced Brownian control problem (or equivalent workload formu- 
lation), that is formulated in the context of our reduced Brownian network. 
For this reduction, we assume the existence of a continuous selection of an 
optimal solution for a minimization problem. In Appendix A. 3 we review 
some related results from convex analysis and describe some sufficient con- 
ditions for the existence of such a continuous selection. Section 7 discusses 
an example that involves new phenomena. 

1.1. Notation and terminology. For a positive integer k, R fc will denote 
A;-dimensional Euclidean space. When k = 1, we shall suppress the super- 
script. For convenience, we define R° to be the real number zero. The Borel 
o"-algebra on this space consists of the empty set and the space R°. These 
conventions concerning R° will be used in treating the degenerate case of 
a zero-dimensional workload process. The nonnegative real numbers will be 
denoted by R+. For x £ R, x + = max(x,0) and x~ = max(— x,0). All vec- 
tors will be assumed to be column vectors unless indicated otherwise. The 
transpose of a vector or matrix will be denoted by a superscript The dot 
product between two vectors x, y € R fc will be denoted by x'y. The Euclidean 
norm of a vector x 6 R fc will be denoted by ||x||. For two sets A and B in 
R fc , and t > 0, we let A + B = {x + y : x € A, y € B} and tA = {tx : x G A}. 
We define the infimum of an empty set of real numbers to be 00. 

For a nonnegative integer k, given a probability space (f2,.F, P), 
a fc-dimensional (stochastic) process defined on this space is a collection 
X = {X(t) : t € R + } of measurable functions X(t) : Q — > R fc , where Q has the 
o"-algebra T and R fc has the Borel a-algebra. All finite-dimensional processes 
appearing in this paper are assumed to have sample paths that are r.c.1.1. 
(right continuous with finite left limits) . If X and Y are two fe-dimensional 
processes defined on a probability space (SI, J 7 , P), then we say that they are 
indistinguishable if 

P(X(t) = Y(t) for all * > 0) = 1. 
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A filtered probability space is a quadruple (fi, T, {Ft}, P) where (fl, T, P) is 
a probability space and {Tt} is a nitration, that is, a family of sub-c-algebras 
of the ex-algebra T indexed by t £ M+ and satisfying .P, C T% whenever 
0<s<t<oo.A fc-dimensional process X = {X(t) : t € R+} defined on such 
a filtered probability space is said to be adapted if for each t > the function 
X(t) : Q — ► M fc is measurable when has the er-algebra T and K fc has its 
Borel er-algebra. 

For a positive integer k, given a filtered probability space (f2, J 7 , {.Ft},P), 
a vector 7 6 a A; x fc symmetric, strictly positive definite matrix S and 
a point x G R fc , an {Pt}-Brownian motion with statistics (7, S) and starting 
point x, is a /c-dimensional adapted process defined on (0,P, {P t },P) such 
that the following hold under P: 

(a) X is a /c-dimensional Brownian motion with continuous sample paths 
that satisfies X(0) =x P-a.s., 

(b) {Xi(t) — ^it, Tt, t > 0} is a martingale for i = 1, . . . , k, and 

(c) {(Xi(t) — jit)(Xj(t) — jjt) — Eijt,Tt,t > 0} is a martingale for i, j = 

In this definition, the filtration {Tt} may be larger than the one generated 
by X; however, for each t > 0, under P, the u-algebra Tt is independent of 
the increments of X from t onward. The latter follows from the martingale 
properties of X . The parameter 7 is called the drift of the Brownian motion 
X and H is called the covariance matrix of X . We adopt the convention that 
a 0-dimensional Brownian motion [with statistics (0,0) and starting point 
= R°], defined on a filtered probability space, is simply a 0-dimensional 
process defined on that space. 

2. Generalized Brownian network. In the following, we shall simply use 
the term "Brownian network," rather than the fuller term "generalized 
Brownian network." The data for a Brownian network consist of: 

(a) positive integers m,n,p, which specify the dimensions of the state 
space, the control space and the control constraint space, respectively, 

(b) a vector z° £ W 71 , a vector 9 6 M m and a symmetric, strictly positive 
definite m x m matrix S, which specify the starting point, drift vector and 
nondegenerate covariance matrix, respectively, for an m-dimensional Brow- 
nian motion, 

(c) an m X n matrix R and a p x n matrix K, which specify the effect of 
controls on the state of the system and constraints on the controls, respec- 
tively, and 

(d) a compact, convex set Z C M. m that has a nonempty interior, which 
specifies the state space. 
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Fix (rn,n,p, z° ,0,T,, R, K, Z) satisfying (a)-(d) above. We now define the 
notion of an admissible control for the Brownian network with this data. This 
definition is formulated in a weak sense, similar to that used for controlled 
stochastic differential equations, see [14]. Before consulting the definition, 
the reader is advised to review the conventions adopted in Section 1.1 con- 
cerning path regularity of stochastic processes, filtered probability spaces 
and associated Brownian motions. All of the processes mentioned in the fol- 
lowing definition are assumed to be defined on the same filtered probability 
space (Q,^F,{^F t },P)- 

Definition 2.1 (Admissible control for the Brownian network). An 
admissible control for the Brownian network is an n-dimensional adapted 
process Y = {Y(t),t > 0} defined on some filtered probability space (Q, T, 
{J~t},P) which supports an m-dimensional adapted process Z and an Tri- 
dimensional {^"tj-Brownian motion X, with statistics (#,£) and starting 
point z°, such that the following two properties hold P-a.s.: 

(i) Z(t) = X(t) + RY(t) € Z for all t > 0, 

(ii) U = {KY(t),t > 0} is nondecreasing and Z7(0) > 0. 

We call Z the state process, (Z, U) the extended state process and X the 
Brownian motion, for the Brownian network under the control Y. 

Remark 2.1. The definition of an admissible control given above is 
slightly different from that used in [11]. The formulation used here is a 
"weak formulation" in the sense that the filtered probability space and the 
Brownian motion are not specified in advance; rather, only the statistical 
properties of the Brownian motion are specified. On the other hand, in [11] 
the Brownian motion and the filtered probability space are given and an 
admissible control must be an adapted process defined on the given space. 
The slightly more general weak formulation adopted here is likely to be par- 
ticularly useful when a pair (Y, X) satisfying the conditions of Definition 2.1 
is obtained as a weak limit from a controlled stochastic processing network. 

Given a continuous function h : Z — ► R and a vector v 6 R n , we associate 
a cumulative cost process £ with any admissible control Y for the Brownian 
network that has state process Z. We let £ be an r.c.1.1. process such that 
almost surely 



The exceptional P-null set on which the above may not hold is necessitated 
by the fact that (i) of Definition 2.1 might only hold almost surely and so 
h(Z) is only well defined almost surely. However, the fact that £ is only 
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uniquely determined up to a P-null set will be of no consequence since our 
ultimate cost functional will involve an expectation under P which does not 
discriminate between indistinguishable processes. 

Remark 2.2. The process v'Y appearing in the last term in (1) may 
have paths that are locally of unbounded variation. Such a complication 
does not occur in [13] since there the corresponding term in the cost process 
is the nondecreasing process c'U, where c is a nonnegative vector. 

We shall make the following assumptions for the remainder of this paper. 

Assumption 2.1. 



On comparing (2) with conditions (i) and (ii) of Definition 2.1, one sees 
that Assumption 2.1 means the following: the system manager has controls 
available to effect an instantaneous displacement in any desired direction at 
any time, although there are state constraints and a possible cost associated 
with such movement. Assumption 2.1 is necessary and sufficient for the 
existence of an admissible control for the Brownian network. The necessity 
is proved in Lemma A. 2 and the sufficiency follows from Theorem A.l. 

Assumption 2.2. 



Assumption 2.2 is used in showing the uniqueness claimed in Lemma 
4.4 below and in proving that the cost functional for our Brownian control 
problem is well defined with values in (— oo, oo] and that this functional has a 
finite lower bound (cf. Lemmas 3.2, A. 3 and Theorem A.l). If the Brownian 
network data arise from a stochastic processing network as in [11], then one 
can show that Assumption 2.2 follows from basic model assumptions (cf. 
Proposition 1 of [11]). Assumption 2.2 can be described as a no-arbitrage 
condition. 

3. Brownian control problem. In this section we define a discounted op- 
timal control problem with infinite planning horizon and cumulative cost 
process £ for our Brownian network. Since our cost functional involves an 
infinite time horizon, some care is needed in its formulation as it is not a 
priori clear that a limit of the finite time horizon discounted cost exists as 
the time horizon recedes to infinity. In fact we prove that for any admissible 
control, almost surely, the limit does exist in (—00,00], with 00 being a pos- 
sible value. Furthermore, we show that the expectation of this limit is well 
defined with a value in (—00,00] (see Lemma 3.2 for the details). 




{Ry.Ky >0,y Gl n } = f 



(3) 



{y G R n : Ky >0,Ry = and v'y < 0} = {0}. 
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Let a > 0. We interpret a as the interest rate for discounting. Given an 
admissible control Y for the Brownian network with extended state process 
(Z, U) and Brownian motion X, for each t > 0, the present value of costs 
incurred over the time interval [0, t] is 

(4) at)=[ e- as d£(s). 

J[o,t] 

By (1), we have that almost surely for all t > 0, 

(5) C(*)= / e~ as h(Z(s))ds+ I e- as d(v'Y)(s). 

Jo J[0,t] 

Here and henceforth, we adopt the usual convention that the contribution 
to the integral in (4) at s = is £(0). The first integral in (5) is well de- 
fined pathwise as a Riemann integral since h is continuous and Z has r.c.1.1. 
paths, so that each path of h(Z) is bounded with at most countably many 
discontinuities on [0,t] (cf. Theorem 7, page 89 of [7]). The second integral 
in (5) is well defined pathwise using a Riemann-Stieltjes integral by Lemma 
A.l. In fact we have the following. 

Lemma 3.1. For each t>0, 

(6) / e" as d(v'Y)(s) = v'Y(0)+ f e~ as d{v'Y)(s) 

J[0,t] J(0,t] 

(7) =a [ e- as v'Y(s)ds + e- at v'Y(t), 

Jo 

where the integrals on the right-hand side above are well defined as Riemann- 
Stieltjes integrals. 

Proof. This follows immediately from the convention about contribu- 
tions at time zero to integrals over the closed time interval [0,t], and from 
Lemma A.l, after observing that s — > e~ as is a continuous function that is 
locally of bounded variation and that each path of v'Y is r.c.1.1. □ 

Almost surely, the first integral in (5) will converge absolutely to a finite 
limit as t — > oo, since Z is bounded almost surely and h is continuous. How- 
ever, we do not know a priori whether the last integral in (5) will converge 
to a limit (finite or infinite) as t — > oo, since we do not have any a priori 
control on the oscillations of v'Y(s) as s — > oo. The almost sure existence 
of a limit for this integral, with values in (—00,00], follows from the next 
lemma which is proved in Appendix A. 2. 
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Lemma 3.2. Almost surely, lim^oo £(£) exists in (—00,00] and satisfies 

(8) limC(t)=/ e- as h(Z(s))ds+ e~ as d{v'Y){s), 

Jo J [0,oo) 

where the first integral in (8) converges absolutely and is bounded in abso- 
lute value by the finite constant sup zGZ \h(z)\/a, and the second integral in 

(8) exists as an improper integral taking values in (—00,00]. In particular, 
almost surely, 

e~ as d(v'Y){s) 

[0,00) 

(9) = lim / e~ as d{v'Y){s) 

rOO 



(10) =a e- as v'Y{s)ds 

Jo 

(11) =a e- as (v'Y(s)) + ds-a e~ as \v'Y \s))~ ds. 



Almost surely, the first integral in (11) takes values in [0, 00] and the last 
integral in (11) has a finite value in [0,oo). This last integral has a finite 
expectation that is bounded by a finite constant not depending on Y . 

Remark 3.1. On comparing (10) with (7), the reader may wonder what 
happened to the last term in (7). As shown in the proof of Lemma 3.2 in 
Appendix A. 2, almost surely one of the following occurs: 

(a) limj^oo e- at v'Y(t) = 0, or 

(b) J °° e- as v'Y(s) ds = 00 and liminf^oo e- at v'Y(t) > -00. 

In either case, the limit as t — > 00 of (7) is equal to the limit as t — ► 00 of 
the first term there. 



Henceforth we shall use C(°°) to denote a random variable that is almost 
surely equal to lim^oo (An exceptional null set on which this random 
variable does not equal the limit or on which the limit may not exist can 
be safely ignored as it will not contribute to the expectation appearing 
in our final cost functional. Similarly, in writing equivalent expressions for 
the expectations of random variables such as C(oo) below, we shall ignore 
null sets on which random variables specified as limits or integrals may be 
undefined.) It follows from the lemma above that the expectation of C(oo) 
exists as a value in (—00, 00]. Accordingly, we adopt the following cost for an 
admissible control Y [with extended state process (Z, U)\ for the Brownian 
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network: 



(12) 



J(Y)^E 
= E 



C(oo)] 



- as h(Z(s))ds 



+ E 



[0,00) 



•d(v'Y)(s) 



Here, by Lemma 3.2, the second last expectation is finite and the last ex- 
pectation is either finite or takes the value 00. Thus, J(Y) € (—00,00]. This 
leads us to make the following definition of a Brownian control problem. 



Definition 3.1 (Brownian control problem — BCP). 
timal value 



Determine the op- 



(13) 



J* 



inf J(Y), 

Y 



where the infimum is taken over all admissible controls Y for the Brown- 
ian network. In addition, if the infimum is attained in (13), determine an 
admissible control Y* that achieves the infimum in (13). We call such a 
control an optimal control for the BCP. On the other hand, if the infimum 
is not attained in (13), for each e > 0, determine an admissible control Y e 
whose cost is within distance e of the infimum. We call such a control Y e 
an e-optimal control for the BCP. 



We show in Theorem A.l that J* is finite, that is, its value lies in 
(—00, 00). 

4. Algebraic manipulations. In this section we perform some manipu- 
lations that will be used in reducing the Brownian network and Brownian 
control problem to a network and control problem of lower dimension. Lem- 
mas 4.1 and 4.2 are analogues of results developed in a somewhat different 
setting in [13]. Lemmas 4.3 and 4.4 relate to the more general cost structure 
assumed here and do not have analogues in [13]. The development given here 
aims to emphasize the geometry of the spaces involved, avoiding choices of 
basis vectors when possible. 

Modifying the notation in [13], let 

(14) M = {y £R n : Ky = 0}, 

(15) n = {Ry:yeM}, 

where Af is mnemonic for null and 1Z is mnemonic for reversible displace- 
ments. Let 1Z 1 - denote the orthogonal complement of 1Z in R m . 



Lemma 4.1. Let 

M = {a € R m : a' R = b'K for some beR p }. 

Then K ± = M. 
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Proof. It suffices to show that 1Z = A4 , the orthogonal complement 
of AA in W n . For this, we note that a G ft if and only if Q) is in the range 

of (_^-)- The latter occurs if and only if (q) is orthogonal to all (|) in the 

orthogonal complement of the range of ( R K ) in R m +P. The last property 

holds if and only if a is in A4^. □ 

Remark 4.1. In view of the above lemma, henceforth we shall use the 
symbols AA and ft -1- interchangeably. 

We now define some additional sets and matrices. Let Af 1 ' denote the 
orthogonal complement of Af in M n . Let /C denote the range of K in MP. 
By restricting its domain, consider K as a linear mapping from Af 1 - into /C. 
This mapping is one-to-one and onto and so has an inverse K > : K, — > N 1 - . 
Thus, K^Ky = y for all y G Af^ and KK^u = u for all u G /C. We can extend 
the definition of the linear mapping to a linear mapping that maps all 
of MP into TV -1 -, for example by defining it to be zero on the orthogonal 
complement KL^ of K in W. We let :W -> R n be such an extension. 
Similarly, R:Af —>7Z is onto and so there is a linear mapping 
such that RE) 5 = 5 for all 5 GlZ. (Note that it^ may only map into M.) We 
can extend R' to a linear mapping defined on all of W 71 into A/" C W 1 , for 
example by defining it to be zero on A4 = 1Z 1 - . This yields a linear mapping 
i?t : R m -»■ M n such that the range of R^ is a subset of M and RR)5 = 6 for 
all 5 G ft. 

Let d be the dimension of A4. If d > 1, let M be the linear mapping 
from lR m onto M d represented by a d x m matrix whose rows are a maximal 
linearly independent set of vectors in A4. If d = 0, let M be the linear 
mapping from R m onto MP (the real number zero). The degenerate case of 
d = can occur in practice and in this case many manipulations simplify. 
For later reference, we let 

(16) W = {Mz:zeZ}. 

Lemma 4.2. There is a linear mapping G from M m into M d such that 

(17) MR = GK. 

Proof. Consider a vector y € M n . Let y and y denote the orthogonal 
projections of y onto TV and Af , respectively, so that y = y + y. Then 
Ry E ft = .M- 1 and so by the definition of M, MRy = 0. (Here, if d > 1, 
denotes the origin in M d , and if d equals zero, then denotes the real number 
zero.) By the definition of K< , since y G y = K^Ky = K^Ky. Thus, 

Mify = MRy = MRK^Ky. 

Since y G M n was arbitrary, it follows that the result holds with G = MRK^ . 
□ 
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Remark 4.2. In general, neither M nor G is unique. In particular, these 
depend on the choice of a basis for Ad. A G that is constructed in the manner 
indicated in the proof of Lemma 4.2 also depends on the choice of K< . For 
Brownian network data arising from a certain class of stochastic processing 
network models, a method for reducing the choices for M and G? to a finite 
set was described in [9]. Following on from this, in [4], two properties of the 
associated workload processes were derived. In a subsequent work, we intend 
to pursue an extension of the method of [9] and to develop properties of the 
associated workload processes for the more general framework of [11]. 

Henceforth we fix a G satisfying the conclusion of Lemma 4.2. However, 
we do not require that G is constructed in the same manner as indicated in 
the proof of Lemma 4.2. In addition to M and G, we shall also need vectors 
7r and k satisfying (18) below. The following lemma guarantees the existence 
of such vectors. 

Lemma 4.3. There is an m- dimensional vector tt and a p- dimensional 
vector k such that 

(18) v' = tt'R + k'K. 
Proof. Let 

■k' = v'R) and «' = v'(I - R)R)K\ 
It suffices to show that for each y £ M. n , 

(19) ir'Ry + k' ' Ky = v'y. 

Fix y € M n . Let y and y denote the orthogonal projections of y onto AT and 
Af , respectively, so that y = y + y. By the definition of K* , 

(20) K^Ky = K ] Ky = y, 
and so 

(21) y = y-y=(I-K^K)y. 

Using the definitions of ir and k, together with (20) and (21), we obtain 

ir'Ry + n'Ky = v'R^Ry + v'(I- W R)K ] Ky 

(22) = v' K ] Ky + v' R ] R(I - K ] K)y 

= v'y + v'R ] Ry. 

We claim that 



(23) 



R ] Ry = y. 
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Assuming that this holds, the desired result (19) then follows immediately 
upon substituting this relation into (22). To see that (23) holds, note that 

(24) R(y - R ] Ry) = Ry - RR^ {Ry) = 

by the definition of R', since Ry G 1Z. Thus, for y' = y — R) Ry we have 
Kyi = 0, Ryf = 0. Then, either y* or — yi satisfies the constraints in the left 
member of Assumption 2.2 and so yi = 0. Hence, (23) holds. □ 

Henceforth, we assume that tt and k are fixed vectors satisfying (18). 
However, as with the choice of G, we do not require that they are constructed 
in the same manner as in the above proof. 

Lemma 4.4. Suppose that x G M. m and u G K, such that Mx = Gu. There 
is a unique y G W 1 such that 

(25) u = Ky and x = Ry, 
given by y = y* where 

(26) y* = y + y, y = K^u, y = R\x-Ry). 
Furthermore, 

(27) v'y* = tt'x + k'u. 

Proof. Let y*,y,y be given by (26). By the definition of y and K\ 
since u G /C, we have y G M 1 ' and Ky = KK^u = u. Furthermore, by the 
definition of R\ y G J\f and so Ky = 0. It follows that Ky* = u. Now, 

M(x - Ry) = Gu- MRy 
= Gu- GKy 
= Gu-Gu 
= 0, 

where we have used the facts that MR = GK and Ky = u. Since the rows 
of M span M, it follows that 

(28) x-Rye M ± = K. 
Thus, since RR J 5 = 5 for all S G 1Z, 

Ry = RR\x -Ry)=x- Ry, 

and so x = Ry* ■ Thus, y = y* satisfies (25). 

To show the uniqueness, suppose that y* is given by (26) and y G M n is 
such that (25) holds. Then, K(y — y*) = and R(y — y*) = 0. Moreover, 
either v'(y — y*) < or v'(y* — y) < 0. Then Assumption 2.2 implies that 
y — y* = 0, which establishes the uniqueness. 

Equation (27) follows by simple algebra, using the fact that y = y* satisfies 
(25) and that tt,k satisfy (18). □ 
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5. Reduced Brownian network. Given data (m, n,p, z°, 6, S, R, K, Z) for 
a Brownian network satisfying the assumptions in Section 2, recall the defi- 
nitions of M, G, IC and W from Section 4. Furthermore, let 

(29) w° = Mz°, $ = M9, r = MUM 1 . 

If d > 1, then T is strictly positive definite, since £ has this property and 
the rows of M are linearly independent. The following defines the notion 
of an admissible control for the reduced Brownian network given the data 
(d,p, w° , #, T, G, IC, W) as described above. It is assumed for this definition 
that all of the processes are defined on the same filtered probability space 
(A, Q, {Qt}, Q)- For the case d = 0, recall our convention that a 0-dimensional 
process (including a Brownian motion) defined on a filtered probability space 
(A, Q, {Qt},Q) is the process defined on A that takes the real value zero for 
all time. 

Definition 5.1 (Admissible control for the reduced Brownian network). 
An admissible control for the reduced Brownian network is a p-dimensional 
adapted process U = {U (t),t> 0} defined on some filtered probability space 
(A, Q,{Qt\,Q) which supports a <i-dimensional adapted process W and a 
d-dimensional {<^}-Brownian motion x> with statistics (?9,r) and starting 
point w°, such that the following two properties hold Q-a.s.: 

(i) W{t) = X (t) + GU(t) G W for all t > 0, 

(ii) U is nondecreasing, U(0) > and U(t) G K, for all t > 0. 

We call W the state process with Brownian motion x f° r the reduced Brow- 
nian network under the control U. 

Remark 5.1. If d = 0, then (i) above reduces to W(t) = for all t > 0, 
and x(t) = for all t > 0. 

In the next two theorems, we describe the relationship between the re- 
duced Brownian network and the Brownian network. 

Theorem 5.1. Suppose that Y is an admissible control for the Brownian 
network with extended state process (Z,U) and Brownian motion X, all 
defined on a filtered probability space (0, {Tt},P). Then on this same 
space, U = KY is an admissible control for the reduced Brownian network 
with state process W = MZ and Brownian motion x = MX . 

Proof. The proof is straightforward on applying M to Definition 2.1 
and using the definitions of W, K, and G. □ 

The following theorem provides a type of converse to the last theorem. 
This result plays an essential role in proving our main result, Theorem 6.1, 
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on the equivalence of the Brownian control problem to the reduced Brownian 
control problem. Recall the definitions of and B) from Section 4. In the 
following, a product extension of a filtered probability space, (A, Q, {Gt}, Q), 
is a filtered probability space, (Q, J 7 , {J~t},P), such that fi = Ax A, T = Q x 
G, Tt = Gt x Gt, P = Q x Q for some filtered probability space (A, Q, {Gt}-, Q)- 
In this case, any process V defined on A can be trivially extended to a process 
defined on f2 by setting 

(30) V(t)(u,Q) = V(t)(uj) for alH>0,wG A,ueA. 

Similarly, any process V defined on A can be trivially extended to a process 
defined on Q. We implicitly assume that such trivial extensions are made 
whenever necessary in the following. 

Theorem 5.2. Let U be an admissible control for the reduced Brownian 
network with state process W and Brownian motion x, oil defined on a 
filtered probability space (A, G ,{Gt},Q) ■ Suppose that Z is a {Gt}-adapted 
m- dimensional process satisfying Q-a.s., 

(31) MZ = W and Z(t) G Z for all t > 0. 

Then there is a product extension (fl,J-,{J-t},P) of the filtered probabil- 
ity space (A,Q,{Q t },Q) such that on this extended space there is an Tri- 
dimensional {Ft} -Brownian motion X with statistics ($,£) and starting 
point z° that satisfies MX = \- On any such extension there is an admis- 
sible control Y for the Brownian network that has extended state process 
(Z,U) and Brownian motion X. Given Z,U,X, the process Y is uniquely 
determined (up to indistinguishability) by 

(32) Y(t)=Y(t)+Y(t), t>0, 
where 

(33) Y(t)=K*U(t), 

(34) Y(t) = Rj(Z(t) — X(t) — RY(t)), 
for each t>0. 

Remark 5.2. The proof of the above theorem involves constructing 
a Brownian motion X from x by adjoining some additional independent 
Brownian motion components. However, as indicated by the theorem, if there 
is already an m-dimensional {^}-Brownian motion X defined on the original 
filtered probability space (A, Q, {Gt}, Q), with statistics (#,£) and starting 
point z° that satisfies x = MX, then one may simply use this Brownian 
motion in constructing Y. 
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Proof of Theorem 5.2. First consider the case where < d < m. 
We show how to extend the probability space to accommodate a suitable 
Brownian motion X. A similar extension is described in Lemma 3.1 of [16]. 
Let N be an (m — d) x m matrix whose rows are a linearly independent set 
of vectors in = 1Z. Then the matrix (^) is a bijection on R m and T = 

(jv)S(^) is a strictly positive definite m x m matrix. The d x d submatrix 
formed by the first d rows and columns of T is the matrix T. Let II denote 
the d x (m — d) submatrix formed by the first d rows and the last m — d 
columns of T (i.e., M'EN'), and let T denote the (m — d)x (m — d) submatrix 
formed by the last (m — d) rows and columns of T (i.e., NUN'). Then 



(35) 



r n 

IT f 



Since the d x d matrix V is real, symmetric and strictly positive definite, 
there is an invertible d x d matrix A such that AA' = T (cf. [6], Theorem 
2.17). The (m — d) x (m — d) matrix T — ITT _1 n (the Schur complement of 
r in T) is also a real, symmetric, strictly positive definite matrix (cf. [6], 
Theorem 2.22), and so there is an invertible (m — d) x (m — d) matrix A 
such that AA' = f - nT - ^. Let = NO and w° = Nz°. 

Let (A, Q, {Gt}, Q) be a filtered probability space, separate from (A, Q, 
{Qt}, Q), on which is defined an (m — d) -dimensional {£/(}-Brownian motion 
B with zero drift, identity covariance matrix and starting point that is the 
origin in R m ~ d . Let CI = A X A, T = Q X G, T t = Q t X Q t for all t > 0, and 
P = Q x Q. Extend the process x defined on (A,Q,{G t },Q) in the trivial 
way so that it is defined on (CI, J 7 , {^i},P). Similarly, extend the process B 
in the trivial way so that it is defined on all of (Cl,J-,{J- t },P). Now, let 

(36) x(t) = n'r -1 (x(*) -•dt- w°) + AB(t) +M + w°. 

Then, (?) is an m-dimensional {^j-Brownian motion with statistics ((|), T) 
and starting point (!L) . Define 



X 



(JO (S 



Then it is straightforward to verify that X is an m-dimensional {J~t\- 
Brownian motion with statistics (0, S) and starting point z° that satisfies 
MX = X . 

Given Z,U,X, the process Y defined by (32)-(34) is an {^}-adapted 
n-dimensional process (with r.c.1.1. paths). Now, P-a.s., (i)-(ii) of Defi- 
nition 5.1 and (31) hold, and for each t > 0, by Lemma 4.4 with x = 
Z(t) - X(t), u = U(t), and noting that 

Mx = MZ(t) - MX(t) = MZ(t) - x(t) = W(t) - X (t) = GU(t) = Gu, 
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we have that Y(t) is the unique element of K n satisfying U(t) = KY{t), 
Z(t) = X(t) +RY(t). It is then easy to verify that Y is an admissible control 
for the Brownian network, with state process (Z, U) and Brownian motion 
X. 

If d = or d = m, the proof is very similar to that above. In particular, 
when d = 0, the linear mapping M and process x are trivial and so are 
ignored in expressions such as (^) and (|). Similarly, when d = m, N and 
X are trivial and are likewise ignored. □ 

6. Reduced Brownian control problem. 

6.1. Equivalent cost structure. The quantity C{t), defined in (4), is inter- 
preted as the discounted cost incurred over the time interval [0,t] under an 
admissible control Y for the Brownian network with extended state process 
(Z,U) and Brownian motion X. Before defining the reduced Brownian con- 
trol problem, we first obtain an equivalent expression for For this, recall 
the definitions of ir and k from Section 4 and define 

(37) g(z) = h(z) + air'z for z G Z. 

Lemma 6.1. Given an admissible control Y for the Brownian network 
with extended state process (Z,U) and Brownian motion X, we have almost 
surely for each t > 0, 

(38) v'Y{t)=ir'(Z(t)-X(t)) + K'U{t) 
and 

f(t)= / e- as g{Z{s))ds+ f e~ as d( K 'U)(s) 



(39) 



[o,t] 



-a I e- as 7r'X(s)ds + e- at ir'{Z(t)-X{t)). 
Jo 

Furthermore, almost surely. 



C(oo)= / e- as g(Z(s))ds+ / e~ as d{K'U){s) 

(40) Jo roo 

-a / e- as ir'X(s)ds, 
Jo 

where the first integral above is absolutely convergent and its absolute value is 
bounded by sup 2g £ \g(z)\/a, the second integral exists as an improper integral 
taking its value in (— oo, oo] and it has an expectation whose value lies in the 
same interval, and the third integral converges absolutely and its absolute 
value has finite expectation. Finally, 



J(Y) = E 

(41) 

+ E 



o 



[0,oo) 



s g(z(s))ds 

'd(n'U)(s) 
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where the first expectation is finite, the second expectation is well defined in 
(—00,00] and I is the finite value defined by 



Here X is mnemonic for integral. 

Proof of Lemma 6.1. Fix an admissible control Y for the Brownian 
network. Consider one of the almost sure realizations such that (i)-(ii) of 
Definition 2.1 and (5) hold. Fix t > 0. Let x = Z(t) - X(t) and u = U(t). 
Then by the properties of Z, U, and since MR = GK, we have 



The equality (38) then follows from Lemma 4.4. To prove (39), use (5), 
Lemma 3.1 and (38) to obtain 



Since Z is r.c.1.1. and g is continuous, each path of g{Z) is r.c.1.1. and bounded 
on [0,t]. The set of points where g(Z) is discontinuous in [0,t] is countable 
and the first integral in (39) is well defined as a Riemann integral (cf. Theo- 
rem 7, page 89 of [7]). Since X is continuous on [0,t], the last integral in (39) 
is also well defined as a Riemann integral. By Lemma A.l, since k'U is r.c.1.1. 
and s — > e~ as is continuous, and of bounded variation on [0, t] , the second 
integral in (39) is well defined using a Riemann-Stieltjes integral. Indeed, 
using the integration-by-parts formula in that lemma, we can rewrite (44) 
in the form of (39). 

In view of Lemma 3.2 and (39), since the last term in (39) tends to zero a.s. 
as t — ► 00 for the proof that (40) holds almost surely, it suffices to show that 
almost surely the integrals on the right-hand side of (40) are well defined in 
the sense described immediately after (40). We now verify these properties. 

Almost surely, Z takes values in Z, a compact set, and g is continuous, 
and then the first integral in (40) converges absolutely and the bound stated 
in the lemma is easily obtained. For the last integral in (40), note that since 
X is a multidimensional Brownian motion with constant drift and fixed 
starting point, there are finite positive constants C\,Ci [depending only on 
the statistics (#,£) and starting point z° of X], such that 



(42) 




(43) 



Mx = MRY{t) = GKY(t) = GU(t) = Gu. 



(44) 




(45) 



E[\\X(s)\\)<C 1 + C 2 s 



for all s > 0. 
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Then using Fubini's theorem we have 



(46) E 



rQO roc 

/ e- as \Tr'X(s)\ds < / e- QS ||7r||(Ci + C 2 s)ds<oc. 
Jo J Jo 



This simultaneously establishes the facts that almost surely the last integral 
in (40) converges absolutely to a finite value and that the expectation of the 
absolute value of the integral is finite. 

The simplest way to see that almost surely the second integral in (40) 
exists as an improper integral taking its value in (—00,00], and that its 
expectation is well defined with value in the same interval, is to leverage the 
fact that similar properties have already been established for the integral 

(47) / e- as d(v'Y){s). 

J[0,oo) 

Indeed, by (38) and integration-by-parts as in Lemma A.l, we have almost 
surely, for each t > 0, 

e~ as cL(k'U)(s) 

[o,t] 

e~ as d(v'Y)(s) - [ e~ as d(-K'{Z -X))(s) 

[0,t] J[0,t] 



f e~ as d{v'Y)(s) - a f e" a V(Z(s) - X(s)) ds 
J[o,t] Jo 



-at 



7r'(Z(t)-X(t)). 



Here, by Lemma 3.2, almost surely, the second last integral above converges 
as t — > 00 to the improper integral J[ooo) e ~° S which has a well- 

defined value in (—00,00] and the expectation of this improper integral is 
well defined with a value in (—00,00]. The last integral above converges al- 
most surely as t — > 00 to a finite limit and the absolute value of this integral 
has finite expectation, since Z is bounded almost surely and X is a Brow- 
nian motion with constant drift. The latter properties can also be used to 
show that the last term above converges almost surely to zero as t — > 00. It 
follows from this that almost surely the second integral in (40) exists as an 
improper integral with value in (—00, 00] and this integral has a well-defined 
expectation in (—00,00]. 

The final claim (41) follows from (40) and the definition of J(Y) as 
£7[£(oc)]. The properties of the various expectations follow from those es- 
tablished above. □ 



The distribution of X , being that of a Brownian motion with prescribed 
statistics, is predetermined and hence uncontrollable. Thus, in terms of de- 
termining an optimal control, the last term I [which depends only on the 
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statistics (9, S) of X and its starting point z°] in the expression (41) for 
the cost functional J(Y) can be ignored. Indeed, we shall use this as one 
simplification in formulating the reduced Brownian control problem. 

6.2. Reduced Brownian control problem. Recall the definition of g from 
(37) and that W = MZ. We define the effective holding cost function 

(48) g(w) =M{g(z):Mz = w,zEZ} for all w 6 W, 
and make the following assumption henceforth. 

Assumption 6.1. The infimum function g:W^M defined by (48) is 
continuous and there is a continuous function ifj : W — > Z such that for each 
w € W, g(ip(w)) = g{w) and Mifj(w) = w. 

We refer the reader to Appendix A. 3 for sufficient conditions that en- 
sure this assumption holds. In particular, the notion of a strictly quasicon- 
vex function is defined in Definition A. 2. As an example, a strictly convex 
function defined on a convex set is strictly quasiconvex. We note that the 
continuous function g is strictly quasiconvex (resp. affine) if and only if h is 
strictly quasiconvex (resp. affine). Then it follows from Lemma A. 5 that suf- 
ficient additional conditions under which Assumption 6.1 holds are that (i) 
the compact, convex set Z is a convex polyhedron, and (ii) the continuous 
function h is strictly quasiconvex or h is affine. 

The following lemma ensures that the cost functional that we plan to use 
for the reduced Brownian network is well defined with values in (—00,00]. 

Lemma 6.2. Given an admissible control U for the reduced Brownian 
network with state process W and Brownian motion x> almost surely for 
each t>0, 

(49) C(*)= t e- as g{W{s))ds+ f e~ as d( K 'U)(s) 

Jo J[0,t] 

is well defined, and 

(50) limC(t)= / e- as g{W{s))ds+ / e~ as d{K'U){s) 

t^oo Jo J [0,oo) 

exists as a value in (—00,00], where the first integral above is absolutely 
convergent and its absolute value is bounded by sup^ eW \g(w)\/a < 00, and 
the second integral exists as an improper integral taking its value in (—00, 00] 
and it has an expectation lying in the same interval. 
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Proof. Fix an admissible control U for the reduced Brownian network. 
Let W denote the associated state process and let \ denote the associated 
Brownian motion. By reasoning similar to that in the proof of Lemma 6.1, 
almost surely for each t > 0, ((t) is well defined, since the first integral in 
(49) exists as a Riemann integral and the second integral is well defined as 

k'U(0)+ [ e~ as d(K'U)(s), 
J(o,t] 

where the last integral above exists as a Riemann-Stieltjes integral, by 
Lemma A.l. 

Now, almost surely, W takes values in the compact set W and g is contin- 
uous, and so the first integral in (50) converges absolutely and is bounded 
in absolute value by sup^ gW \g(w)\/a. 

For the second integral in (50), since almost surely, W lives in W, we 
can define an adapted process Z such that Z = ip(W) almost surely. Then Z 
satisfies the hypotheses of Theorem 5.2, and so one can extend the underlying 
filtered probability space to one with a filtration denoted by {J~t} on which 
there is defined an m-dimensional {^f}-Brownian motion X with statistics 
(#,£) and starting point z° such that MX = x (cf. Theorem 5.2), and for 
any such extension, Y defined by (32)— (34) is an admissible control for the 
Brownian network with extended state process (Z, U). Then, it follows from 
applying Lemma 6.1 to this Y, that almost surely the second integral in (50) 
is well defined as an improper integral taking values in (—00,00] and that it 
has an expectation lying in the same interval. □ 

Henceforth, we let C(oo) denote a random variable that is almost surely 
equal to lim^oo C(i). It follows from Lemma 6.2 that the expectation of 
C(oo) exists as a value in (—00,00]. We adopt the following cost for an 
admissible control U (with associated state process W), for the reduced 
Brownian network: 



J(U) = E 
(51) =E 



CM] 



00 







e - as g(W(s))ds 



+ E 


\f c 




y [0,00) 



d(K'U)(s) 



Here, by Lemma 6.2, the second last expectation is finite and the last ex- 
pectation is either finite or takes the value 00. Thus, J(U) € (—00,00]. We 
define the reduced Brownian control problem as follows (this is sometimes 
alternatively called the equivalent workload formulation) . 

Definition 6.1 (Reduced Brownian control problem — RBCP). Deter- 
mine the optimal value 

(52) J* = mfJ(U), 
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where the infimum is taken over all admissible controls U for the reduced 
Brownian network. In addition, if the infimum is attained in (52), determine 
an admissible control U* that achieves the infimum. We call such a control 
an optimal control for the RBCP. On the other hand, if the infimum is not 
attained in (52), for each e > 0, determine an admissible control U e whose 
cost is within distance e of the infimum. We call such a control U e an e- 
optimal control for the RBCP. 

Remark 6.1. In order to derive the RBCP from the BCP, one needs 
to choose linear mappings M, G, and vectors w, k, as in Section 4. Given 
the data for a BCP, these then determine the data for a reduced Brownian 
network (cf. Section 5) and the cost (51) for the RBCP. The function g is 
given by the optimization in (48), where g [given by (37)] depends on the 
cost function h, the interest rate a and the vector tt, and the feasible region 
depends on M and the state space Z for the Brownian network. 

In the sense of the following theorem, our Brownian control problem is 
equivalent to the reduced Brownian control problem. Here, for ease of ter- 
minology, we shall use the term e-optimal control with e = for an optimal 
control. 

Theorem 6.1. The optimal value J* of the Brownian control problem 
[BCP) and the optimal value J* of the reduced Brownian control problem 
(RBCP) are related by 

(53) J* = J*+1, 

where I is defined by (42) for some m- dimensional Brownian motion X 
with statistics (9, S) and starting point z°. Fix e > 0. If Y [with extended 
state process (Z, U) and Brownian motion X] is an e-optimal control for the 
Brownian control problem, then U = KY (with state process W = MZ and 
Brownian motion x — MX) is an e-optimal control for the reduced Brown- 
ian control problem. Conversely, if U (with state process W and Brownian 
motion x) is o,n e-optimal control for the reduced Brownian control problem, 
then after setting Z = ip(W) almost surely and enlarging the filtered prob- 
ability space as in Theorem 5.2 to one whose filtration is denoted by {J~t} 
and which accommodates an m- dimensional {Tt}- Brownian motion X with 
statistics (9,T<) and starting point z° such that MX = x, we have that the 
process Y defined by (32) -(34) is an e-optimal control [with extended state 
process (Z, U) and Brownian motion X] for the Brownian control problem. 

Proof. Suppose that Y [with extended state process (Z, U) and Brow- 
nian motion X] is an admissible control for the Brownian network. Set 
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W = MZ and x = MX. Then, by Theorem 5.1, U (with state process W 
and Brownian motion x) is an admissible control for the reduced Brownian 
network. By the definition (48) of g, we have that, almost surely, 



and so 



(54) 



J* < J(U) 



E 



< E 



g(W(t))<g(Z(t)) 

as g(W(s))ds\ +E 
as g(Z(s))ds - 



for all t > 0, 



J(Y)+2, 



[0,oo) 



[0,oo) 



S d(K'U){s) 

'd(K'U)(s) 



where we have used the definition of J* in the first line, the definition of 
</([/) in the second line and (41) in the last line. By taking the infimum over 
all admissible controls Y for the Brownian network, we see that 



(55) 



J* <J* + 1. 



Now, suppose that U (with state process W and Brownian motion x) is an 
admissible control for the reduced Brownian network. Since, almost surely, 
W lives in W, we can define an adapted process Z such that Z = ip(W) 
almost surely. Then Z satisfies the hypotheses of Theorem 5.2, and so, in 
the manner described in Theorem 5.2, one can extend the underlying filtered 
probability space to one with a filtration denoted by {J~t} on which there 
is defined an m-dimensional {^}-Brownian motion X with statistics £) 
and starting point z° such that MX = x, and for any such extension, Y 
defined by (32)-(34) is an admissible control for the Brownian network with 
extended state process (Z,U). Then by the definition of tp, we have almost 
surely for all t > 0, 



(56) 



g{Z{t))=g^{W{t)))=g{W{t)), 



and so using (41) again we have 

e - as g(W{s))ds 



J{U) = E 
(57) = E 



s g(z(s))ds 



+ E 
+ E 



J{Y)+1>J* +1. 



[0,oo) 



[0,oo) 



! d(K'U)( S ) 



- aS d{K'U){ S ) 



By taking the infimum over all admissible controls U for the reduced Brow- 
nian network, we obtain that 



(58) 



J*>J*+ 1. 
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Fig. 1. An example with two servers working in parallel. 



Combining (55) with (58) yields that 

(59) J* = J*+1. 

Given e > 0, if Y [with extended state process (Z, U)] is an e-optimal 
control for the BCP, then it follows from (54) and (59) that 

(60) J(U)< J(Y)+2< J*+e+l = J*+e, 

and hence that U is an e-optimal control for the RBCP. 

Similarly, given e > 0, if U (with state process W) is an e-optimal control 
for the RBCP and an admissible control Y for the Brownian control problem 
is derived from U as described above, then it follows from (57) and (59) that 

(61) J{Y) = J(U)-2<J* + e-l = J*+e, 
and so Y is an e-optimal control for the BCP. □ 



7. An example. Let us consider the stochastic processing network por- 
trayed in Figure 1, which was discussed in Sections 3 and 7 of [11]. The 
following four paragraphs are excerpted from pages 1126-1127 of [11]. 

We imagine that units of flow are discrete; those units will be called "jobs" 
and processing resources will be called "servers." Here we have two servers 
(represented by the circles in Figure 1) and two job classes that are stored 
in separate buffers (represented by the open-ended rectangles in Figure 1) 
as they await processing. 

For each job class i = 1, 2 the average arrival rate Aj, expressed in jobs per 
hour, is as shown in Figure 1. There are a total of six "processing activities" 
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in our example, the first four of which are portrayed in Figure 1. (The 
numbering of activities is arbitrary.) Each activity j = 1,...,4 consists of 
a particular server processing jobs from a particular buffer, the associated 
average service rate being jjij jobs per hour (see Figure 1). With "activity 
levels" expressed in server hours, one may alternatively say that fi\, . . . ,//4 
each represent an average rate of material flow per unit of activity. 

In addition to the processing activities described above, there are two 
activities that we use to represent input control capabilities: activities 5 
and 6 correspond to the system manager ejecting jobs from buffers 1 and 
2, respectively, which we assume can be done at any time without penalty. 
However, such "disposal" is irreversible, and thus it deprives the system 
manager of whatever value might have been derived from processing the 
jobs ejected. 

With regard to system economics, let us suppose that each activity j = 
1, . . . , 4 generates value at an average rate of yj hundred dollars per completed 
job, where y\ = 1, y% = 1, y% = 2 and < 7/4 < For each of these activities, 
then, the average value generated per unit of activity (i.e., per server-hour 
devoted to the activity) is Vj = yjf^j hundred dollars. Assuming that there 
is neither direct cost nor direct benefit associated with activities 5 and 6, 
we then have the value rate vector 

(62) u = (1,1, 1,1/4, 0,0)' where < v 4 < 3/2. 

Using this information and various other model assumptions (including 
Poisson arrivals and exponential service time distributions), a generalized 
Brownian network was derived in Section 7 of [11] to approximate the sys- 
tem pictured in Figure 1. The state space dimension of that Brownian net- 
work is m = 2 and its control space dimension is n = 6, while its control 
constraint space has dimension p = 5. The matrices R and K appearing in 
the descriptions of Z and U are 

1 \ 1 0" 

1 3 lj ' 

1 1 " 
110 
-1 , 
-1 
.0 -1. 

and the parameters of the underlying (uncontrolled) Brownian motion X 
are 



(64) 



"0" 




"0" 




"2.2 


" 





, = 










1.6. 



(63) 



R 



K 
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Finally, the state space of the generalized Brownian network derived in [11] 
is Z = [0, b] x [0, b], where b > is large, and the holding cost function is 

(65) h(z) = a\z\ + 02-22 where 01,02 > 0. 

Given these data and an interest rate a > for discounting, the system 
manager's dynamic control problem is formally approximated by the Brow- 
nian control problem (BCP) set forth at the end of Section 3. It would be 
wasteful to repeat all of the reasoning advanced in [11] to support that ap- 
proximation, but a few salient points are essential for interpretation of the 
analysis to follow. First, if one considers the deterministic fluid analogue of 
our example, one finds that the optimal processing strategy uses only activ- 
ities 1, 2 and 3. (Activity 4, although it processes class 2 jobs quickly, does 
not generate enough economic value per job processed to justify its use.) 
When stochastic variability is introduced, that same mix of activities 1, 2 
and 3 constitutes the "nominal processing plan," but one or both servers 
may experience occasional idleness due to starvation (i.e., lack of work to 
do) and, in addition, activities 4 through 6 may be used sparingly to re- 
duce buffer contents when holding costs threaten to become excessive. The 
stochastic process Z% in our approximating Brownian network model cor- 
responds to the contents of buffer i in scaled units (i = 1,2), and the five 
components of the nondecreasing process U are interpreted (in scaled units) 
as follows: U\ corresponds to cumulative unused capacity for server 1; U2 
corresponds to cumulative unused capacity for server 2; U3 corresponds to 
cumulative time devoted to activity 4 by server 1, and U4 and U5 correspond 
to the cumulative number of jobs ejected from buffers 1 and 2, respectively. 

The key step in solving the BCP for our example, and the primary focus 
of this section, is derivation of a reduced Brownian control problem (RBCP) 
using the recipe laid out in Sections 5 and 6. To determine a matrix M that 
defines the workload process for our RBCP, we begin with the following 
observation: a vector j/GR 6 satisfies Ky = if and only if y2 + 1/3 = and 
Ui = Va = Vh = V% = 0. Thus the space J\f of all such y is spanned by the 
vector yf = (0, 1, —1, 0, 0, 0)', implying that the space 1Z is spanned by Ry^ = 
(-5, 1)'. Now Ai = 1Z 1 - by Lemma 4.1, so Ai is the one-dimensional subspace 
of R 2 spanned by the row vector 

(66) M=(2,l). 

The space K is all of M 5 since K has full row rank. 

As noted in Section 4, one can take G to be any matrix satisfying MR = 
GK, and it and k to be any pair of vectors satisfying tt'R + k'K = v' . In our 
current example, given M, the choice 



(67) 



G= (2, 1,-1, -2,-1) 
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is unique, and the system of linear equations (18) that determines the seven- 
dimensional row vector (it 1 , k') has rank 6; arbitrarily setting tt\ = 1 gives 

(68) vr' = (l,i) and k' = (0, §, § - v 4 , 1, ±). 

Substituting (66) in (16), we identify the state space of our reduced Brow- 
nian network as W = [0, 3b], where b > is large. Also, according to (29) and 
(64), the one-dimensional Brownian motion \ that appears in the reduced 
Brownian network (see Section 5) has initial state w° = 0, drift parameter 
•& = and variance parameter T = MY.M' = 10.4. In the reduced Brownian 
network we have a one-dimensional workload process W that almost surely 
satisfies the main system equation 

(69) W(t) = x(t) + GU(t) G [0, 36] for all t > 0, 

where the five-dimensional control vector U has nonnegative and nonde- 
creasing components. In our reduced Brownian control problem (see Section 
6), the objective is to choose a control U so as to minimize 



(70) J{U) = E 



oo 



e~ as g(W(s))ds 



+ E 


\f e 




J[0,oo) 



where g is defined in terms of the holding cost function h, the interest rate a 
and the vector 7r, via (37) and (48). Recall from (37) that g(z) = h(z) + ait l ' z. 
For our example, one finds that ir' = hM, implying that 

(71) tt'z = \w for all z G M 2 such that Mz = w. 

Thus the vector z = ip(w) that achieves the infimum in (48) is the same z 
that minimizes h(z) = a\z\ + a,2z\ subject to the constraints Mz = w and 
z£Z. 

Since Z = [0, b] x [0, b] is a convex, compact polyhedron, and the contin- 
uous function g is strictly convex (hence strictly quasiconvex) , it follows 
from Lemma A. 5 that g is continuous and ip : W — > Z is continuous. It is 
straightforward to also verify that g is convex using the convexity of Z and 
g. In fact, one can explicitly solve the optimization problem (48) for g and 
ijj. The specification of tp(w) breaks into three cases depending on whether 
< w < b, b < w < 2b or 2b < w < 3b. For example, when < w < 6, 

(72) ^ 1 {w) = [—^ — )w and ik(w) = ( — — )w 

and 

(73) g( W ) = g(if;( W )) = QlQ2 W 2 + ^-W. 

4a 2 + ai 2 

For b < w <2b, to ensure that tp2(w) = w — 2tpi(w) < b, ip\{w) is the max- 
imum of the value specified in (72) and Finally, for 2b < w < 3b, to 
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ensure in addition that ip\{w) < b, one takes ipi( w ) to be the minimum of b 
and the value given by the formula for ipi(w) used in the case b < w <2b. 
In all cases, ^{w) =w — 2ipi(w). 

The RBCP described immediately above is a one-dimensional "singular" 
control problem, and after one additional simplification the analysis of Har- 
rison and Taksar [12] can be invoked for its solution. Using the notation of 
this paper, the problem solved in [12] is the following. A controller contin- 
uously monitors the evolution of a Brownian motion x that has arbitrary 
mean and strictly positive variance and initial state in a given finite interval 
W. The controller chooses two nondecreasing, nonnegative processes L\ and 
L2, each nonanticipating with respect to \i and the "state of the system" 
W is defined via 



The controller is obliged to keep W within the given interval W and the 
objective to be minimized is 



where g is convex and £\ and £2 are constants satisfying £\ + £2 > 0. Although 
the problem specified in [12] requires all controls to be nonanticipating, 
the analysis performed there (which relies on Ito's formula) is still valid if 
one only requires that the controls L\ and L2 are adapted to a filtration 
with respect to which the Brownian motion x minus its drift process is a 
martingale. This observation enables us to apply the results of [12] to our 
setting where a "weak" formulation is used for admissible controls. The 
paper [12] also assumes that the last two expectations in (75) are finite for 
admissible controls L\,L2- In fact, it is sufficient for the analysis of [12] 
that the last two terms in (75) are finite. In particular, if £- t = 0, then this 
condition for the term involving ti is automatically satisfied. In our RBCP 
example, it will turn out that t\ = 0, £2 > 0. Thus, for good controls having 
finite cost, the term in (75) involving £2 > will be finite. 

To see how our reduced Brownian control problem can be further reduced 
to the one described in the previous paragraph, it remains to show how 
the five modes of singular control (nondecreasing processes U\ 1 ...,U^) in 
our RBCP can be reduced to two modes of singular control (nondecreasing 
processes L\ and L2). It turns out that three of the five modes of control in 
the RBCP can be eliminated as follows. 

From (67) and (69) we see that the system manager has available two 
means of increasing the workload level W, namely, by increasing either U\ or 
U2, and those control actions have associated direct costs per unit of control 



(74) 



W{t) = X {t) + L l {t)-L 2 {t) 



for t > 0. 



(75) 



E 
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of k\ = and K2 = i, respectively. In other words, for the same increase 
in workload level, there is no direct cost for using U\, whereas there is a 
positive direct cost associated with using JJ%. Thus, in choosing an optimal 
control, one will only ever use U\ to increase the workload and one will never 
use Ui- Consequently, in translating our RBCP to the form (74)— (75), we 
can define the singular control L\ = 2U\ and associate with L\ the cost rate 
£i = 0. 

On the other hand, we see from (67) and (69) that the system manager can 
instantaneously decrease the workload W by increasing the control Uf. for 
any k = 3, 4, 5. For a given decrease in workload level, the preferred means of 
achieving this is to use the control k £ {3, 4, 5} for which Kk/\Gf.\ is minimal. 
That is, one chooses the control mode having least direct cost per unit of 
workload reduction. Recall from (62) that < v 4 < 3/2 by assumption. If 1 < 
i>4 < 3/2, then we see from (67) and (68) that increasing U3 is the preferred 
means of effecting downward displacement of workload, the associated cost 
per unit of displacement being K3/IG3I = 3/2 — V4 < 1/2. In that case one 
can set U4 = U5 = in the RBCP, define L2 = U3, and associate with L 2 the 
cost rate £2 = 3/2 — V4, thereby reducing the RBCP to the form (74)-(75). 
The interpretation is as follows: when the workload W gets high enough to 
motivate a costly downward displacement (see below), the system manager 
will insert the fast but not very lucrative activity 4, meaning that server 1 
devotes some of its time to processing class 2 jobs while server 2 readjusts the 
mix of its activities correspondingly to effect the desired workload reduction. 

Alternatively, if < V4 < 1, then increasing U4 or increasing U5 is the pre- 
ferred means of effecting a downward displacement, so one can set U3 = 
in the RBCP and define L2 = IU4 + U5, the associated cost per unit of dis- 
placement being I2 = K4/IG4I = K5/IG5I = 1/2. This is interpreted to mean 
that, as a means of reducing workload, the system manager is indifferent 
between rejecting class 1 arrivals and rejecting class 2 arrivals. If 1)4 = 1, 
then all three means of reducing workload are equally attractive: we define 
L2 = Us + 2U4 + U5 and £2 = 1/2 in that case. 

When the results of [12] are applied to our example, one has the following: 
there exists an optimal policy that imposes a lower reflecting barrier at 
W = 0, where L\ increases, and an upper reflecting barrier at W = b* > 0, 
where L2 increases. (Of course, the optimal barrier height b* depends on the 
drift and variance parameters for the uncontrolled Brownian motion \i 011 
the reduced holding cost function g, on the costs of effecting upward and 
downward displacements of W and on the interest rate a. It can be shown 
that b* I as V4 T |.) The workload level at any time t > is by definition 
W(t) = MZ(t) = 2Z 1 (t) + Z 2 (t). Translating the optimal solution of the 
RBCP into an optimal solution for the BCP involves setting Z(t) = tp(W(t)) 
at each time t > 0. 
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In terms of desired behavior for the original stochastic processing net- 
work, one intuitively interprets the above solution to mean that the system 
manager should strive to use only activities 1, 2 and 3 when the workload 
is strictly greater than zero and less than b* , to incur idleness only at server 

1 (because U2 = 0) and then only when the workload is near zero, and to 
effect rapid downward displacement of the workload (by whatever means is 
preferred, as discussed above) whenever the workload is at or above the level 
b* . Furthermore, the system manager needs to switch the attention of server 

2 between buffer 1 and buffer 2 (see Figure 1) so as to keep \Z(t) —tp(W(t))\ 
small. To achieve these aims (at least approximately), it is likely that vari- 
ous dynamic priorities and thresholds could be employed. The formulation 
and investigation of asymptotic optimality of such policies is a significant 
separate undertaking that is not pursued here. 

APPENDIX 

A.l. Real analysis lemma. The following real analysis lemma will be 
used several times in manipulating costs. 

Lemma A.l. Let /:R+ —> R be a continuous function that is locally of 
bounded variation and let g : R + — ► R be a right continuous function on R + 
that has finite left limits on (0, 00). Then for each t>0, 

(76) / g(s)df(s) and f f(s)dg(s) 
J(o,t] J(o,t] 

are well defined as Riemann-Stieltjes integrals and they are related by the 
following integration-by-parts formula: 

(77) / f(s) dg(s) + / g(s) df(s) = f(t)g(t) - /(0)g(0). 
J(o,t] J(o,t] 

Proof. See Theorem 18, page 278 and Theorem 8, page 265 of [7]. □ 

A. 2. Behavior of the cost process over the infinite time horizon. Through- 
out this section, the data (m,n,p,z ,9,H,R,K,2) for a Brownian network 
is fixed and satisfies (a)-(d) of Section 2. We first prove that Assumption 
2.1 is necessary for the existence of an admissible control for the Brownian 
network. 

Lemma A. 2. Suppose that there is an admissible control Y , with Brow- 
nian motion X, for the Brownian network (cf. Definition 2.1). Then As- 
sumption 2.1 must hold. 
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Proof. For a proof by contradiction, suppose that Assumption 2.1 does 
not hold. Let 

V = {Ry:Ky >0,y GlP} 

and x G M. m \V. Since V is closed, there is 5 > such that the distance from 
x to V is greater than 25. For each e > 0, let 

(78) B(x,e) = {zeR m :\x- z\<e}. 
Then, 

(79) B(x,25)nV = 0. 

Since X is a Brownian motion with nondegenerate covariance matrix, for 
each t > 0, 

(80) P(-X(t)eB(tx,tS))>0. 

Since Y satisfies (i)-(ii) of Definition 2.1 P-a.s., it follows from (80) that for 
each t > 0, 

(81) P(RY(t) £Z + B(tx, tS),KY(t) > 0) > 0, 
and hence 

(82) P(i?y(i)t" 1 G r l Z + B(x, 5),KY(t)t~ 1 > 0) > 0. 

Since Z is compact, t~ l Z will be in B(0,5) for all t sufficiently large and 
for such t, 

t- 1 Z + B(x,5)cB(x,25), 

and so 

P{RY{t)r l G B(x,26),KY(t)r 1 > 0) > 0. 

It follows, on setting y = Y(t)t~ l for a suitable realization, that there is 
y€l" such that 

Ry£B(x,25), Ky>0. 
Then Ry G P(x, 25) n V, which contradicts (79). □ 

For the remainder of this section, we assume that Assumptions 2.1 and 
2.2 hold. Below we prove Lemma 3.2 and we show that the optimal value 
J* of the cost for the Brownian control problem is finite. First we establish 
some useful preliminary lemmas. 
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Lemma A. 3. Let T> be a nonempty compact set in R m . The following 
set is either empty or it is a nonempty compact set: 

(83) {yeR n :Ky>0,RyeV,v'y<0}. 
In either case, 

(84) ini{v'y : Ky >0,Ry£V,y£ R n } > -co. 

Proof. Fix a nonempty compact set T> in R m . It suffices to prove the 
first statement in the lemma, for if the set in (83) is empty, then the infimum 
in (84) lies in [0, oo]; on the other hand, if the set in (83) is nonempty 
and compact, then the infimum in (84) is the same as the infimum of the 
continuous function y — > v'y over this compact set, which is finite. 

To prove that the set in (83) is either empty or a nonempty compact 
set, we give a proof by contradiction. For this, suppose that the set in (83) 
is nonempty and unbounded. Since T> is a nonempty compact set, we can 
find a nonempty, bounded, convex polyhedron V that contains T>. Since the 
set in (83) is unbounded, the following set which contains it must also be 
unbounded: 

(85) {y €E R n : Ky > 0, Ry € V, v'y < 0}. 

The set above is an unbounded convex polyhedron and so it must contain 
a ray, that is, it contains a set of the form {a + Xb : A > 0} where a, b £ R n 
and 6^0 (cf. [2], Proposition 3.2.2). Then Rb = since V is bounded and 
XRb £V - Ra for all A > 0. Also, Kb > -Ka/X for all A > 0. By letting 
A — » oo, we see that Kb > 0. Similarly, v'b < —v'a/X for all A > and on 
letting A — > oo we obtain v'b < 0. Thus, b ^ satisfies Rb = 0, Kb > and 
v'b < 0. This contradicts Assumption 2.2 and so the desired result follows. 
□ 

Corollary A.l. Let 

(86) 7 = - mi{v'y :Ky>0, \\Ry\\ <l,y€ M. n }, 

(87) r? = sup{||y|| : Ky > 0, pfy|| < l,v'y <0,j/6 M n }. 
TTien 7 and 77 are finite nonnegative constants and for each A > 0, 

(88) mf{v'y :Ky>0, \\Ry\\ <X,y£R n } = -A 7 , 

(89) sup{||y|| : Ky > 0, \\Ry\\ < A, v'y <0,y£R n } = Xrj. 

PROOF. The set {y G R n : Ky > 0, \\Ry\\ < I, v'y < 0} is nonempty, be- 
cause it contains y = 0. Then by Lemma A. 3, with D={i£ R m : \\x\\ < 1}, 
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this nonempty set is compact. It follows that 7 and 77 are nonnegative and 
finite. For A > 0, on setting y = y/X, we obtain 

ini{v'y : Ky > 0, < A, y 6 M n } 

= A mf{v'y : Ky > 0, \\Ry\\ <ljGl n } = -A 7 

and 

sup{||y|| : Ky > 0, \\Ry\\ < A, v'y < 0,y E R n } 

= Asup{||y|| :Ky > 0, < l,v'y <0,yeR n } = \ V , 

as desired. □ 

The following lemma is key to the proof of Lemma 3.2 which is given 
further below. 

Lemma A. 4. Let Y be an admissible control for the Brownian network 
with extended state process (Z,U) and Brownian motion X. Almost surely, 
either: 

(i) lim t ^ 00 e- at (v'Y(t)) + = 0, or 

(ii) / °° e- as {v f Y(s)) + ds = 00. 

Proof. Let f2o be an ^"-measurable set of probability 1 such that on 
Qo, properties (i)-(ii) of Definition 2.1 hold and limt^ 00 e~ Qi |[X(t)|| = 0. In 
the following we assume that a fixed realization oj from has been chosen. 
To simplify the notation, we suppress explicit mention of u. We consider 
two cases: either the limit in the left member of (i) does not exist in [0, 00] 
[Case (a)] or it does exist [Case (b)]. 

Case (a). Suppose that 

(90) limsupe- Q VY(i)) + + liminf e -trf (t/y(t)) + . 

Then there are constants < a < b < 00 such that 

(91) liminf e~ at {v'Y {t)) + <a<b< limsupe~ a \v'Y(t)) + . 

Let ro = and inductively define for each integer n > 0, 

(92) r 2n+1 = inf{i > r 2n : e - at {v'Y (t)) + > 6}, 

(93) r 2n+2 = mi{t > T 2n+1 : e~ a \v'Y (t))+ < a}. 
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Then by (91), r n < oo for all n, and r n — > oo as n —* oo, since the paths of 
Y are r.c.1.1. For each n > 0, let 

A n y = y(r 2n+2 )-y(r 2n+1 ), 

A n X = X(r 2n+2 ) - X(r 2n+1 ), 
A n Z = Z(T 2n+2 ) - Z(T 2n +i)- 
Then, from (92)-(93), and the choice of a realization in we have 

v'A n Y = v'Y(T 2n +2) ~ v'Y(T 2n+ l) 

(94) < e Qr2 "+ 2 a-6e QT2 ™ +1 , 

(95) K(A n Y) = U(T 2n+2 ) - U(r 2n+1 ) > 0, 

(96) R(A n Y) = A n Z - A n X. 
Let e > such that r , b ■, > 1 and let 

(97) 5 = - log 



a V (1 + 
If 

(98) T 2n+2 — T 2n+ \ > 5 for infinitely many n, 

then 



/"OO fT2n + 2 

(99) / e' as (v'Y(s)) + ds>Y, ads>5-a- 

JO n JT2n + l 



OO = OO. 



Conversely, if 

(100) T 2n+2 — r 2n+ i < 5 for all but finitely many n, 

then, by (94) and (97), we have for all n sufficiently large, 

v'A n Y < e aT2n+1 (ae a(T2n + 2 " T2 " +l) - b) 
< e aT2n+1 ( — b 



(101) ^ e {IT! 

eb 

— c aT2n + l 

1 + e 

When (101) holds, by (95)-(96), we have 

A n Ye{y£M n :Ky>0,\\Ry\\ < \\A n Z\\ + ||A re X||, v'y < 0}. 
and then by Corollary A.l, 

l|A n y||<77(l|A n Z|| + ||A,„X||). 
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Thus, when (100) holds we have that for all n sufficiently large [since both 
sides of (101) are negative], 

< \v' £\ n Y\e~ aT2n+1 



l + e 

< ^IMKIIA^II + ||A n Al)e- aT2 " +1 

< |U| ( W^ Z W + W X ( T 2n+2) || e aS ||X(r 2n+1 ; 



g«T"2n+l g«T"2n+2 gC«T"2n + l 

Using the compactness of Z, the fact that r n — > oo as n — > oo, and the 
asymptotic behavior of the Brownian motion X on Oq, we see that the last 
expression above tends to zero as n — > oo. However, this implies that 

eb 

<0, 



l + e 



which is a contradiction since e > and b > 0. It follows that only (98) can 
hold and then (99) holds and (ii) follows. 

Case (b). Suppose that (90) does not hold, that is, 

limsupe- at (v'Y{t)) + =limmfe- at (v'Y{t)) + =£, 

for some E € [0, oo]. 

If £ G (0, oo], then there are £' € (0,£) and r € [0, oo) such that 

e~ at (v'Y(t)) + >£' for all i>r, 

and then for all t>r, 

t e- as {v'Y(s)) + ds>{t-T)£', 



where the last expression tends to co as t — > oo, which implies that (ii) holds. 
If £ = 0, then 

lim e- a \v'Y(t)) + = 0, 

t— >oo 

and (i) holds. 

Since either Case (a) or Case (b) must hold, this completes the proof. □ 

Proof of Lemma 3.2. Using the fact that almost surely, for all s > 0, 
Z(s) € Z, a compact set, and the fact that h is continuous, we see that 
almost surely the integral 

coo 

e- as h(Z(s))ds 
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converges absolutely and is bounded by sup zeZ \h(z)\/a. So it suffices to 
focus on the behavior as t — > oo of 



[o,t] 



(102) 



a 



a 



] d(v'Y){s) 



e- as v'Y(s)ds + e- at v'Y(t) 



e- as (v'Y(s)) + ds + e- at {v'Y(t)Y 



a 



(v'Y(s))-ds 



-octf.J 



v'Y{t))-. 



Now, almost surely, Z(s) G Z, RY(s) = Z(s) — X(s) and KY(s) > for each 
s > 0, and then by Corollary A.l we have 



(103) 



e- as (v'Y{s))-ds 



< 



\Z(s)\\ + \\X(s)\\)d S , 



where the integral is finite almost surely since Z is a compact set and X 
is a multidimensional Brownian motion with constant drift starting from 
z° . Indeed, there are finite positive constants C\, C2 (depending only on the 
statistics of X, its starting point z° and Z) such that 



(104) 
Hence, 



(105) 



E[\\Z(s)\\ + ||X(s)||] < d + C 2 s for all s > 0. 
- as (v'Y(s))-ds 



E 





= 7 

<7 







*E[\\Z{s)\\ + \\X{s)\\]ds 



e- as (C 1 + C 2 s)ds< 00. 



Similarly, almost surely, 
(106) 



(v'Y(t))-<je~ at (||Z(f)|| + ||X(t)||)-0 



as t 



00. 



Thus, almost surely, the last line in (102) converges as t — > 00 to 



-a 



'(v'Y(s))-ds 



where the last integral has a finite expectation that is bounded by the con- 
stant in the last line of (105), which does not depend on Y. The remaining 
part of (102) to consider is 

(107) a C e- as (v'Y(s)) + ds + e- at (v'Y(t)) + . 
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For this, we observe from Lemma A. 4 that almost surely, either: 
(a) lim^oo e~ at (v'Y(t)) + = and then as t -> oo, (107) tends to 



a 



or 



(b) 



'(i/Y(s)) + ds€ [0,oo] 



, (v'Y(s)) + ds = oo 



and then as t — > oo, (107) tends to oo, regardless of the behavior of the 
nonnegative quantity e~ at (v'Y(t)) + . 

Thus, in either case, almost surely, as t — > oo, (107) converges to 



'(v'Y(s)) + ds £ [0, oo]. 



Combining all of the results above yields the desired result. □ 

We now obtain lower and upper bounds on the value J* of the Brownian 
control problem. 



Theorem A. 1. J* e (-00,00). 

Proof. For any admissible control Y [with extended state process (Z, U) 
and Brownian motion X] for the Brownian network, the second last term 
in the expression (12) for the cost J(Y) is bounded in absolute value by 
sup zeZ \h(z)\/a, which does not depend on Y. By Lemma 3.2 and its proof 
[especially (105)], there are finite positive constants C\,C2 [depending only 
on 2, the Brownian motion statistics (9, S) and the starting point z°\ such 
that the last term in the expression (12) satisfies 



E 



(108) 



[0,00) 
= aE 

> -aE 



as d{v'Y){s) 

POD 

/ e~ as v'Y{s) ds 
Jo 

poo 

3 / e- as (v'Y(s))' ds 

>-a 7 / e- as {Ci + C 2 s)ds. 
Jo 



It follows that J(Y) is bounded below by a fixed finite constant for all 
admissible controls Y. Hence, J* > —00. 

To prove that J* < 00, it suffices to demonstrate that there is an admis- 
sible control Y such that J(Y) < 00. Assumption 2.1 ensures the existence 
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of such a control. One such control can be constructed as follows. Choose a 
nonempty open ball B lying in the interior of Z. Let e^, . . . ,e( m ) be unit 
vectors parallel to each of the positive coordinate axes in M m . By Assump- 
tion 2.1, for each i E {1, . . . ,m}, we can find y(' l \y( m +^ g R n such that 

(109) Ry®=e®, Ky®>0, 

(110) Ry( m+ ^ = -e«, Ky( m+i ^ > 0. 

Let r > denote the radius of B. For each point z on the boundary dB of 
B, the distance from z to the center of B is r. Given cc € K m with ||x|| < r, 
we have 

m m 

x = ^ x . e w = ^ (x +_ xr)e w j 

i=l i=l 

where xf, < r for all i. Then, for 

m 

(in) y = Y J {4y {t} +*7y {m+i) l 

i=l 

we have Ry = x and Ky > 0, where 

2m 

(H2) IMI<r£||y«|| = <7(r). 

i=l 

Given an {^j-Brownian motion X, with statistics (#,£) and starting point 
z°, defined on a filtered probability space (Q,J 7 ,{J 7 t},P), we define an ad- 
missible control Y with state process Z and Brownian motion X as follows. 
This process Y is a pure jump process. At time zero, define Y(0) to be a 
fixed vector in W 1 such that KY(0) > and RY(0) = c° — z° where c° is the 
center of the ball B (the existence of such a vector follows from Assumption 
2.1). Let Z(0) = c°. From time zero onward, whenever Z is in the interior of 
B, let Z have the same increments as X and do not let Y change. Whenever 
Z approaches the boundary of B, at the time that it would have reached the 
boundary, let it jump immediately to the center c° of the ball B, and then 
continue on from there using the increments of X. If z is the position on the 
boundary that Z would have reached, then the jump in Y that is used to 
produce the jump to c° is given by (111) with x = c° — z. It is straightfor- 
ward to see that this informal description of the construction of Y and Z, so 
that (i)-(ii) of Definition 2.1 hold P be made formal using a suit- 

able increasing sequence of stopping times. We leave this to the interested 
reader. Under any admissible control for the Brownian network, including 
the one just described, the second last expectation in (12) is bounded by 
sup z£Z \h(z)\/a, which is finite since Z is bounded and h is continuous. Un- 
der the admissible control just described, using a regeneration argument, 
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it is straightforward to show that the last expectation in (12) is bounded 
above by 

(H3) hi (imo) 

where [3 = E[e~ aT ], r is the first time that a Brownian motion with statistics 
(6, £) starting from c° hits the boundary of B. By the continuity of the paths 
of X, (3 < 1 and so the expression in (113) is finite. It follows that J(Y) for 
the aforementioned control is finite and hence this provides a finite upper 
bound for J*. □ 

A. 3. Continuous selection. In this section we develop some results con- 
cerning continuity of the optimal value and of an optimizer as functions 
of certain constraints in optimization problems. These results provide suf- 
ficient conditions for Assumption 6.1 of Section 6 to be satisfied. Here 
m,n,p, R, K, Z are fixed and satisfy the properties specified in (a), (c) and 
(d) of Section 2. The linear mapping M and set W are defined as in Section 
4. In particular, 

(114) W = {Mz:zeZ}. 

Given a continuous function g : Z — > R, for each weW, consider the fol- 
lowing minimization problem: 

(115) minimize g(z) subject to Mz = w,z 6 Z. 

By the definition of W, for each w € W, the feasible set of solutions 

(116) $(w) = {zeZ:Mz = w} 

is nonempty. Since Z is compact and convex, and the mapping defined by 
M is continuous and linear, $(u;) is compact and convex for each w € W. It 
follows from the compactness of $>(w) and the continuity of g that for each 
ioSW the function g achieves its minimum value 

(117) g(w) = inf {g(z) : z € $(w)} 
on $>(w). For each w 6 W, the set of minimizers 

(118) *(w) = {ze$(w):g(z)=g(w)} 

is nonempty; however, the set *$>(w) may contain more than one point. 

For reducing the Brownian control problem, we shall be interested in 
conditions under which g is continuous on W and there is a continuous 
function tp : W — > Z such that ij)(w) G ^(w) for each w S W. Such a contin- 
uous function ip is called a continuous selection for It seems difficult to 
give necessary and sufficient conditions for the existence of such a continu- 
ous selection. Below we give sufficient conditions for the continuity of g and 
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the existence of a continuous selection for We also show by example that 
a continuous selection need not always exist. For this we recall the follow- 
ing definitions and properties from the theory of quasiconvex functions. For 
more details, we refer the reader to [1]. 

Definition A.l. A real-valued function / defined on a convex set C C 
R m is quasiconvex if its lower-level sets 

(119) L(f,a) = {zeC:f(z)<a} 
are convex for every a G M. 

In fact, a real- valued function / defined on a convex set C C M. m is qua- 
siconvex if and only if 

(120) /(As« + (1 - \)z®) < max{/( Z W),/( Z ( 2 ')} 

for all z^\z^ G C and < A < 1 (cf. [1], Theorem 3.1). This motivates the 
following definition. 

Definition A. 2. A real- valued function / defined on a convex set C C 
M. m is strictly quasiconvex if 

(121) /(A*« + (1 - X)zM) < max{/(z«), f(z^)} 
for all < A < 1 and , z^> G C satisfying ^ z^ . 

The following properties are straightforward to verify. First, a strictly 
convex function defined on a convex set is strictly quasiconvex there. Second, 
a strictly quasiconvex function / on a convex set C attains its infimum over 
C at no more than one point in C. 

Lemma A. 5. Consider a continuous function g:Z^R. Suppose that: 

(i) the compact, convex set Z is a convex polyhedron, and 

(ii) g is strictly quasiconvex or g is affine. 

Then the infimum function <?:W — ► R defined by (117) is continuous and 
there is a continuous function ip : W — > Z such that ip(w) G $(w) and g(ijj(w)) = 
g(w) for each w G W, that is, ip is a continuous selection for the set-valued 
function ^ defined by (118). 

Proof. Suppose that Z is a convex polyhedron, that is, 
Z = {zeR m : J a® < b^ for * = 1, ... , £}, 
for some aS*' G R m , el, i = 1, . . . ,£, and a positive integer £. 
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If there is a continuous selection tp for ^, then since g is also continuous, 
g(w) = g(tp(w)) is a continuous function of w G W. Thus, it suffices to prove 
the existence of such a continuous selection tp. 

First, suppose that g is strictly quasiconvex on the convex polyhedron Z. 
For each w G W, ^f(w) contains a single point and 

$(w) = {zeR m :z'a {i) < for i = l,...,£;Mz = w} 

is a nonempty, convex polyhedron. It follows from the latter and Corollary 
II. 3.1 of [5] that lim^oo $(wk) = $(w) for any sequence {wk}'^L 1 in W that 
converges to w G W. The desired result then follows from Corollary 1.3.4 of 
[5]. (In that corollary the star notation implicitly assumes that there is a 
unique minimizer.) 

On the other hand, if g is affine, that is, g(z) = z'a + b for all z G M m for 
some a G M m and b G R, then the minimization problem (115) is equivalent 
to a linear program of the form considered in [3]. In this case, for a given 
w G W, ^(if) need not be a singleton. However, $>(«;) is a compact set for 
each (and hence at least one) w G W. It then follows from Theorem 2 of [3] 
that the set-valued mapping \F from W into subsets of W 71 is continuous. 
Also, ^(w) is convex for each w G W. Then, as noted by Bohm [3], it follows 
by Michael's selection theorem (cf. [15], pages 188-190) that one can make 
a continuous selection ip from ^. □ 

The following concrete example shows that a continuous selection may 
fail to exist if g is quasiconvex but not strictly quasiconvex. 

Example A.l. We shall describe a continuous quasiconvex function 
g-.M 2 — > R_|_ by describing the level sets of g. For each r > 0, the set on 
which g takes the value r is the union of the following four line segments: 



(122) 


{zel 


i 2 


:z 2 =r,- 


•r < z\ < r}, 


(123) 


{zGl 


i 2 


: z\ = -r, 


— r < Z2 < r}, 


(124) 


{zGl 


i 2 


: z 2 = -r, 


— r < zi < r 2 }, 


(125) 


{zGl 


i 2 


:z= (r 2 , 


-r)+t(r-r 2 ,2r),0<t<l} 



Some level sets of the function g are drawn in Figure 2 for the values of r = 
0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0. One can verify that g is a continuous 
quasiconvex function on M 2 . However, it is not strictly quasiconvex, since 
its level sets contain line segments. Given w G M, consider the following 
optimization problem: 

(126) minimize g(z) subject to z\ =w, \z% \ < 2, \z 2 \ < 2. 

We focus on optimizers of this problem when w is near 1. For w = 1, the 
set of minimizing solutions is {z G M 2 : z\ = 1, — 1 < z 2 < 1}. For each value 
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■4 



2 



2 



Fig. 2. Some level sets of the continuous quasiconvex function g described in Example 



of w in (0, 1) U (1,2], the function g has a unique minimum in the set {z € 
R 2 : z\ = w, \z\\ < 2, 1 22 1 < 2}. Moreover, as io approaches 1 from below, this 
minimizer approaches the point (1,1), whereas for w approaching 1 from 
above, this minimizer approaches the point (1,-1). It follows that there 
cannot be a continuous selection of a minimizing solution z as a function of 
w near w = 1. 
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